"Finding My Piece in That Puzzle": A Qualitative Study Exploring How Medical Students at Four U.S. Schools Envision Their Future Professional Identity in Relation to Health Systems.

PURPOSE: Health systems science (HSS) curricula equip future physicians to improve patient, population, and health systems outcomes (i.e., to become "systems citizens"), but the degree to which medical students internalize this conception of the physician role remains unclear. This study aimed to explore how students envision their future professional identity in relation to the system and identify experiences relevant to this aspect of identity formation. METHOD: Between December 2018 and September 2019, authors interviewed 48 students at 4 U.S. medical schools with HSS curricula. Semistructured interviews were audiorecorded, transcribed, and analyzed iteratively using inductive thematic analysis. Interview questions explored how students understood the health system, systems-related activities they envisioned as future physicians, and experiences and considerations shaping their perspectives. RESULTS: Most students anticipated enacting one or more systems-related roles as a future physician, categorized as "bottom-up" efforts enacted at a patient or community level (humanist, connector, steward) or "top-down" efforts enacted at a system or policy level (system improver, system scholar, policy advocate). Corresponding activities included attending to social determinants of health or serving medically underserved populations, connecting patients with team members to address systems-related barriers, stewarding health care resources, conducting quality improvement projects, researching/teaching systems topics, and advocating for policy change. Students attributed systems-related aspirations to experiences beyond HSS curricula (e.g., low-income background; work or volunteer experience; undergraduate studies; exposure to systems challenges affecting patients; supportive classmates, faculty, and institutional culture). Students also described future-oriented considerations promoting or undermining identification with systems-related roles (responsibility, affinity, ability, efficacy, priority, reality, consequences). CONCLUSIONS: This study illuminates systems-related roles medical students at 4 schools with HSS curricula envisioned as part of their future physician identity and highlights past/present experiences and future-oriented considerations shaping identification with such roles. These findings inform practical strategies to support professional identity formation inclusive of systems engagement.

Health systems science education: The new post-Flexner professionalism for the 21st century.

The foundations of medical education have drawn from the Flexner Report to prepare students for practice for over a century. These recommendations relied, however, upon a limited set of competencies and a relatively narrow view of the physician's role. There have been increasing calls and recommendations to expand those competencies and the professional identity of the physician to better meet the current and future needs of patients, health systems, and society. We propose a framework for the twenty-first century physician that includes an expectation of new competency in health systems science (HSS), creating 'system citizens' who are effective stewards of the health care system. Experiential educational strategies, in addition to knowledge-centered learning, are critically important for students to develop their professional identity as system citizens working alongside interprofessional colleagues. Challenges to HSS adoption range from competing priorities for learners, to the need for faculty development, to the necessity for buy-in by medical schools and their associated health care systems. Ultimately, success will depend on our ability to articulate, encourage, support, and evaluate system citizenship and its impact on health care and health care systems.

Medical Student Attitudes toward USMLE Step 1 and Health Systems Science - A Multi-Institutional Survey.

Phenomenon: Because of its importance in residency selection, the United States Medical Licensing Examination Step 1 occupies a critical position in medical education, stimulating national debate about appropriate score use, equitable selection criteria, and the goals of undergraduate medical education. Yet, student perspectives on these issues and their implications for engagement with health systems science-related curricular content are relatively underexplored. Approach: We conducted an online survey of medical students at 19 American allopathic medical schools from March-July, 2019. Survey items were designed to elicit student opinions on the Step 1 examination and the impact of the examination on their engagement with new, non-test curricular content related to health systems science. Findings: A total of 2856 students participated in the survey, representing 23.5% of those invited. While 87% of students agreed that doing well on the Step 1 exam was their top priority, 56% disagreed that studying for Step 1 had a positive impact on engagement in the medical school curriculum. Eighty-two percent of students disagreed that Step 1 scores should be the top item residency programs use to offer interviews. When asked whether Step 1 results should be reported pass/fail with no numeric score, 55% of students agreed, while 33% disagreed. The majority of medical students agreed that health systems science topics were important but disagreed that studying for Step 1 helped learn this content. Students reported being more motivated to study a topic if it was on the exam, part of a course grade, prioritized by residency program directors, or if it would make them a better physician in the future. Insights: These results confirm the primacy of the United States Medical Licensing Examination Step 1 exam in preclinical medical education and demonstrate the need to balance the objectives of medical licensure and residency selection with the goals of the broader medical profession. The survey responses suggest several potential solutions to increase student engagement in health systems science curricula which may be especially important after Step 1 examination results are reported as pass/fail.

Health Systems Science in Medical Education: Unifying the Components to Catalyze Transformation.

Medical education exists in the service of patients and communities and must continually calibrate its focus to ensure the achievement of these goals. To close gaps in U.S. health outcomes, medical education is steadily evolving to better prepare providers with the knowledge and skills to lead patient- and systems-level improvements. Systems-related competencies, including high-value care, quality improvement, population health, informatics, and systems thinking, are needed to achieve this but are often curricular islands in medical education, dependent on local context, and have lacked a unifying framework. The third pillar of medical education-health systems science (HSS)-complements the basic and clinical sciences and integrates the full range of systems-related competencies. Despite the movement toward HSS, there remains uncertainty and significant inconsistency in the application of HSS concepts and nomenclature within health care and medical education. In this Article, the authors (1) explore the historical context of several key systems-related competency areas; (2) describe HSS and highlight a schema crosswalk between HSS and systems-related national competency recommendations, accreditation standards, national and local curricula, educator recommendations, and textbooks; and (3) articulate 6 rationales for the use and integration of a broad HSS framework within medical education. These rationales include: (1) ensuring core competencies are not marginalized, (2) accounting for related and integrated competencies in curricular design, (3) providing the foundation for comprehensive assessments and evaluations, (4) providing a clear learning pathway for the undergraduate-graduate-workforce continuum, (5) facilitating a shift toward a national standard, and (6) catalyzing a new professional identity as systems citizens. Continued movement toward a cohesive framework will better align the clinical and educational missions by cultivating the next generation of systems-minded health care professionals.

Incorporating Quality Improvement into Resident Education: Structured Curriculum, Evaluation, and Quality Improvement Projects.

BACKGROUND: Quality Improvement (QI) is essential for improving health care delivery and is now a required component of neurosurgery residency. However, neither a formal curriculum nor implementation strategies have been established by the Accreditation Council for Graduate Medical Education. METHODS: We describe our experience with implementing a formal QI curriculum, including structured didactics and resident led group-based QI projects. Course materials and didactics were provided by the Mayo Quality Academy. Participants were required to take a 30-question multiple-choice exam to demonstrate basic proficiency in QI methods following completion of didactic. An anonymous survey also was performed to elicit feedback from course participants. RESULTS: All of the 40 student participants (17 residents) were able to demonstrate basic proficiency in QI methods on a standardized exam upon course completion. Of the 9 attempted QI projects, 7 were completed, with 5 of those resulting in sustained process changes. The majority of participants felt formal training improved confidence in QI processes and was a valuable professional tool for their careers. CONCLUSIONS: A formal didactic curriculum and practical application of QI methodologies adds value to resident training. Further, it has the potential to positively impact practice. Consideration should be given to adopting a formal QI curriculum by other neurosurgery departments and perhaps standardization on national level.

Imprinting on Clinical Rotations: Multisite Survey of High- and Low-Value Medical Student Behaviors and Relationship with Healthcare Intensity.

BACKGROUND: Physician behaviors are important to high-value care, and the learning environment medical students encounter on clinical clerkships may imprint their developing practice patterns. OBJECTIVES: To explore potential imprinting on clinical rotations by (a) describing high- and low-value behaviors among medical students and (b) examining relationships with regional healthcare intensity (HCI). DESIGN: Multisite cross-sectional survey PARTICIPANTS: Third- and fourth-year students at nine US medical schools MAIN MEASURES: Survey items measured high-value (n = 10) and low-value (n = 9) student behaviors. Regional HCI was measured using Dartmouth Atlas End-of-Life Chronic Illness Care data (ratio of physician visits per decedent compared with the US average, hospital care intensity index, ratio of medical specialty to primary care physician visits per decedent). Associations between regional HCI and student behaviors were examined using unadjusted and adjusted (controlling for age, sex, and year in school) logistic regression analyses, using median item ratings to summarize reported engagement in high- and low-value behaviors. KEY RESULTS: Of 2623 students invited, 1304 (50%) responded. Many reported trying to determine healthcare costs (1085/1234, 88%), but only 45% (571/1257) reported including cost details in case presentations. Students acknowledged suggesting tests solely to anticipate what their supervisor would want (1143/1220, 94%), show off their ability to generate a broad differential diagnosis (1072/1218, 88%), satisfy curiosity (958/1217, 79%), protect the team from liability (938/1215, 77%), and build clinical experience (533/1217, 44%). Students in higher intensity regions reported significantly more low-value behaviors: each one-unit increase in the ratio of physician visits per decedent increased the odds of reporting low-value behaviors by 20% (OR 1.20, 95% CI 1.04-1.38; P = 0.01). CONCLUSIONS: Third- and fourth-year medical students report engaging in both high- and low-value behaviors, which are related to regional HCI. This underscores the importance of the clinical learning environment and suggests imprinting is already underway during medical school.

Attitudes toward cost-conscious care among U.S. physicians and medical students: analysis of national cross-sectional survey data by age and stage of training.

BACKGROUND: The success of initiatives intended to increase the value of health care depends, in part, on the degree to which cost-conscious care is endorsed by current and future physicians. This study aimed to first analyze attitudes of U.S. physicians by age and then compare the attitudes of physicians and medical students. METHODS: A paper survey was mailed in mid-2012 to 3897 practicing physicians randomly selected from the American Medical Association Masterfile. An electronic survey was sent in early 2015 to all 5,992 students at 10 U.S. medical schools. Survey items measured attitudes toward cost-conscious care and perceived responsibility for reducing healthcare costs. Physician responses were first compared across age groups (30-40 years, 41-50 years, 51-60 years, and > 60 years) and then compared to student responses using Chi square tests and logistic regression analyses (controlling for sex). RESULTS: A total of 2,556 physicians (65%) and 3395 students (57%) responded. Physician attitudes generally did not differ by age, but differed significantly from those of students. Specifically, students were more likely than physicians to agree that cost to society should be important in treatment decisions (p < 0.001) and that physicians should sometimes deny beneficial but costly services (p < 0.001). Students were less likely to agree that it is unfair to ask physicians to be cost-conscious while prioritizing patient welfare (p < 0.001). Compared to physicians, students assigned more responsibility for reducing healthcare costs to hospitals and health systems (p < 0.001) and less responsibility to lawyers (p < 0.001) and patients (p < 0.001). Nearly all significant differences persisted after controlling for sex and when only the youngest physicians were compared to students. CONCLUSIONS: Physician attitudes toward cost-conscious care are similar across age groups. However, physician attitudes differ significantly from medical students, even among the youngest physicians most proximate to students in age. Medical student responses suggest they are more accepting of cost-conscious care than physicians and attribute more responsibility for reducing costs to organizations and systems rather than individuals. This may be due to the combined effects of generational differences, new medical school curricula, students' relative inexperience providing cost-conscious care within complex healthcare systems, and the rapidly evolving U.S. healthcare system.

Value-Added Activities in Medical Education: A Multisite Survey of First- and Second-Year Medical Students' Perceptions and Factors Influencing Their Potential Engagement.

PURPOSE: To describe attitudes of first- and second-year U.S. medical students toward value-added medical education, assess their self-reported desire to participate in value-added activities, and identify potentially modifiable factors influencing their engagement. METHOD: The authors conducted a cross-sectional survey of first- and second-year students at nine U.S. medical schools in 2017. Survey items measured students' attitudes toward value-added medical education (n = 7), desire to participate in value-added activities (n = 20), and factors influencing potential engagement (n = 18). RESULTS: Of 2,670 students invited to participate, 1,372 (51%) responded. Seventy-six percent (1,043/1,368) moderately or strongly agreed they should make meaningful contributions to patient care. Students' desire to participate was highest for patient care activities approximating those traditionally performed by physicians, followed by systems improvement activities and lowest for activities not typically performed by physicians. Factors increasing desire to participate included opportunities to interact with practicing physicians (1,182/1,244; 95%), patients (1,177/1,246; 95%), and residents or fellows (1,166/1,246; 94%). Factors decreasing desire to participate included making changes to the health care system (365/1,227; 30%), interacting with patients via phone or electronic communication (410/1,243; 33%), and lack of curricular time (634/1,233; 51%). CONCLUSIONS: First- and second-year medical students agree they should add value to patient care, but their desire to participate in value-added activities varies depending on the nature of the tasks. Medical schools may be able to increase students' desire to participate by enabling face-to-face interactions with patients, embedding students in health care teams, and providing dedicated curricular time.

High-Value, Cost-Conscious Communication Skills in Undergraduate Medical Education: Validity Evidence for Scores Derived from Two Standardized Patient Scenarios.

INTRODUCTION: Training in high-value, cost-conscious care (HVCCC) is increasingly being incorporated into medical school curricula, but students may have limited opportunities to engage patients in HVCCC conversations. The aim of this study was to develop two standardized patient scenarios with associated checklists, hypothesizing that resulting scores would allow for valid formative assessments of HVCCC communication skills. METHODS: Scenarios were designed to generate a less-is-more conversation (in response to a patient requesting an unnecessary test) and a shared decision-making conversation (in response to a patient choosing between multiple effective treatment options). Checklists were developed by experts and informed by the existing literature. Validity evidence was collected from content, response process, internal structure, relations to other variables, and consequences of testing. RESULTS: Ninety-three third-year medical students participated during 2014-2015. Mean checklist scores were 79% (SD = 18, Cronbach α = 0.72) and 72% (SD = 13, Cronbach α = 0.62) for the less-is-more and shared decision-making scenarios, respectively. Checklist scores correlated with global ratings of performance (r = 0.65 and 0.54, respectively, both P < 0.001), and overall interrater reliability was good (r = 0.66). Checklist scores discriminated between higher and lower performers (discrimination indices of 0.84 and 0.65, respectively, both P < 0.001). Most students (83/90, 92%) agreed that the session improved their HVCCC communication skills. CONCLUSIONS: This study provides validity evidence supporting the use of scores derived from two standardized patient scenarios for formative assessment of HVCCC communication skills among third-year medical students. These scenarios can help equip students with practical, patient-centered strategies for promoting value in clinical encounters.

Science of health care delivery milestones for undergraduate medical education.

BACKGROUND: The changing healthcare landscape requires physicians to develop new knowledge and skills such as high-value care, systems improvement, population health, and team-based care, which together may be referred to as the Science of Health Care Delivery (SHCD). To engender public trust and confidence, educators must be able to meaningfully assess physicians' abilities in SHCD. We aimed to develop a novel set of SHCD milestones based on published Accreditation Council for Graduate Medical Education (ACGME) milestones that can be used by medical schools to assess medical students' competence in SHCD. METHODS: We reviewed all ACGME milestones for 25 specialties available in September 2013. We used an iterative, qualitative process to group the ACGME milestones into SHCD content domains, from which SHCD milestones were derived. The SHCD milestones were categorized within the current ACGME core competencies and were also mapped to Association of American Medical Colleges' Entrustable Professional Activities (AAMC EPAs). RESULTS: Fifteen SHCD sub-competencies and corresponding milestones are provided, grouped within ACGME core competencies and mapped to multiple AAMC EPAs. CONCLUSIONS: This novel set of milestones, grounded within the existing ACGME competencies, defines fundamental expectations within SHCD that can be used and adapted by medical schools in the assessment of medical students in this emerging curricular area. These milestones provide a blueprint for SHCD content and assessment as ongoing revisions to milestones and curricula occur.

Science of health care delivery as a first step to advance undergraduate medical education: A multi-institutional collaboration.

Physicians must possess knowledge and skills to address the gaps facing the US health care system. Educators advocate for reform in undergraduate medical education (UME) to align competencies with the Triple Aim. In 2014, five medical schools and one state university began collaborating on these curricular gaps. The authors report a framework for the Science of Health Care Delivery (SHCD) using six domains and highlight curricular examples from each school. They describe three challenges and strategies for success in implementing SHCD curricula. This collaboration highlights the importance of multi-institutional partnerships to accelerate innovation and adaptation of curricula.

Ways to Write a Milestone: Approaches to Operationalizing the Development of Competence in Graduate Medical Education.

PURPOSE: To identify approaches to operationalizing the development of competence in Accreditation Council for Graduate Medical Education (ACGME) milestones. METHOD: The authors reviewed all 25 "Milestone Project" documents available on the ACGME Web site on September 11, 2013, using an iterative process to identify approaches to operationalizing the development of competence in the milestones associated with each of 601 subcompetencies. RESULTS: Fifteen approaches were identified. Ten focused on attributes and activities of the learner, such as their ability to perform different, increasingly difficult tasks (304/601; 51%), perform a task better and faster (171/601; 45%), or perform a task more consistently (123/601; 20%). Two approaches focused on context, inferring competence from performing a task in increasingly difficult situations (236/601; 29%) or an expanding scope of engagement (169/601; 28%). Two used socially defined indicators of competence such as progression from "learning" to "teaching," "leading," or "role modeling" (271/601; 45%). One approach focused on the supervisor's role, inferring competence from a decreasing need for supervision or assistance (151/601; 25%). Multiple approaches were often combined within a single set of milestones (mean 3.9, SD 1.6). CONCLUSIONS: Initial ACGME milestones operationalize the development of competence in many ways. These findings offer insights into how physicians understand and assess the developmental progression of competence and an opportunity to consider how different approaches may affect the validity of milestone-based assessments. The results of this analysis can inform the work of educators developing or revising milestones, interpreting milestone data, or creating assessment tools to inform milestone-based performance measures.

Health Systems Science Curricula in Undergraduate Medical Education: Identifying and Defining a Potential Curricular Framework.

PURPOSE: The authors performed a review of 30 Accelerating Change in Medical Education full grant submissions and an analysis of the health systems science (HSS)-related curricula at the 11 grant recipient schools to develop a potential comprehensive HSS curricular framework with domains and subcategories. METHOD: In phase 1, to identify domains, grant submissions were analyzed and coded using constant comparative analysis. In phase 2, a detailed review of all existing and planned syllabi and curriculum documents at the grantee schools was performed, and content in the core curricular domains was coded into subcategories. The lead investigators reviewed and discussed drafts of the categorization scheme, collapsed and combined domains and subcategories, and resolved disagreements via group discussion. RESULTS: Analysis yielded three types of domains: core, cross-cutting, and linking. Core domains included health care structures and processes; health care policy, economics, and management; clinical informatics and health information technology; population and public health; value-based care; and health system improvement. Cross-cutting domains included leadership and change agency; teamwork and interprofessional education; evidence-based medicine and practice; professionalism and ethics; and scholarship. One linking domain was identified: systems thinking. CONCLUSIONS: This broad framework aims to build on the traditional definition of systems-based practice and highlight the need for medical and other health professions schools to better align education programs with the anticipated needs of the systems in which students will practice. HSS will require a critical investigation into existing curricula to determine the most efficient methods for integration with the basic and clinical sciences.

Role Modeling and Regional Health Care Intensity: U.S. Medical Student Attitudes Toward and Experiences With Cost-Conscious Care.

PURPOSE: To examine medical student attitudes toward cost-conscious care and whether regional health care intensity is associated with reported exposure to physician role-modeling behaviors related to cost-conscious care. METHOD: Students at 10 U.S. medical schools were surveyed in 2015. Thirty-five items assessed attitudes toward, perceived barriers to and consequences of, and observed physician role-modeling behaviors related to cost-conscious care (using scales for cost-conscious and potentially wasteful behaviors; Cronbach alphas of 0.82 and 0.81, respectively). Regional health care intensity was measured using Dartmouth Atlas End-of-Life Chronic Illness Care data: ratio of physician visits per decedent compared with the U.S. average, ratio of specialty to primary care physician visits per decedent, and hospital care intensity index. RESULTS: Of 5,992 students invited, 3,395 (57%) responded. Ninety percent (2,640/2,932) agreed physicians have a responsibility to contain costs. However, 48% (1,1416/2,960) thought ordering a test is easier than explaining why it is unnecessary, and 58% (1,685/2,928) agreed ordering fewer tests will increase the risk of malpractice litigation. In adjusted linear regression analyses, students in higher-health-care-intensity regions reported observing significantly fewer cost-conscious role-modeling behaviors: For each one-unit increase in the three health care intensity measures, scores on the 21-point cost-conscious role-modeling scale decreased by 4.4 (SE 0.7), 3.2 (0.6), and 3.9 (0.6) points, respectively (all P < .001). CONCLUSIONS: Medical students endorse barriers to cost-conscious care and encounter conflicting role-modeling behaviors, which are related to regional health care intensity. Enhancing role modeling in the learning environment may help prepare future physicians to address health care costs.

Priority Areas and Potential Solutions for Successful Integration and Sustainment of Health Systems Science in Undergraduate Medical Education.

Educators, policy makers, and health systems leaders are calling for significant reform of undergraduate medical education (UME) and graduate medical education (GME) programs to meet the evolving needs of the health care system. Nationally, several schools have initiated innovative curricula in both classroom and workplace learning experiences to promote education in health systems science (HSS), which includes topics such as value-based care, health system improvement, and population and public health. However, the successful implementation of HSS curricula across schools is challenged by issues of curriculum design, assessment, culture, and accreditation, among others. In this report of a working conference using thematic analysis of workshop recommendations and experiences from 11 U.S. medical schools, the authors describe seven priority areas for the successful integration and sustainment of HSS in educational programs, and associated challenges and potential solutions. In 2015, following regular HSS workgroup phone calls and an Accelerating Change in Medical Education consortium-wide meeting, the authors identified the priority areas: partner with licensing, certifying, and accrediting bodies; develop comprehensive, standardized, and integrated curricula; develop, standardize, and align assessments; improve the UME to GME transition; enhance teachers' knowledge and skills, and incentives for teachers; demonstrate value added to the health system; and address the hidden curriculum. These priority areas and their potential solutions can be used by individual schools and HSS education collaboratives to further outline and delineate the steps needed to create, deliver, study, and sustain effective HSS curricula with an eye toward integration with the basic and clinical sciences curricula.

Science of Health Care Delivery: An Innovation in Undergraduate Medical Education to Meet Society's Needs.

The purpose of this special article is to describe a new, 4-year Science of Health Care Delivery curriculum at Mayo Clinic School of Medicine, including curricular content and structure, methods for instruction, partnership with Arizona State University, and implementation challenges. This curriculum is intended to ensure that graduating medical students enter residency prepared to train and eventually practice within person-centered, community- and population-oriented, science-driven, collaborative care teams delivering high-value care. A Science of Health Care Delivery curriculum in undergraduate medical education is necessary to successfully prepare physicians so as to ensure the best clinical outcomes and patient experience of care, at the lowest cost.

Quality Improvement Education for Health Professionals: A Systematic Review.

Effective quality improvement (QI) education should improve patient care, but many curriculum studies do not include clinical measures. The research team evaluated the prevalence of QI curricula with clinical measures and their association with several curricular features. MEDLINE, Embase, CINAHL, and ERIC were searched through December 31, 2013. Study selection and data extraction were completed by pairs of reviewers. Of 99 included studies, 11% were randomized, and 53% evaluated clinically relevant measures; 85% were from the United States. The team found that 49% targeted 2 or more health professions, 80% required a QI project, and 65% included coaching. Studies involving interprofessional learners (odds ratio [OR] = 6.55; 95% confidence interval [CI] = 2.71-15.82), QI projects (OR = 13.60; 95% CI = 2.92-63.29), or coaching (OR = 4.38; 95% CI = 1.79-10.74) were more likely to report clinical measures. A little more than half of the published QI curricula studies included clinical measures; they were more likely to include interprofessional learners, QI projects, and coaching.

Developmental Screening and Follow-up by Nurses.

OBJECTIVE: The Ages and Stages Questionnaire (ASQ) and Modified Checklist for Autism in Toddlers (M-CHAT) are commonly used screening tools for developmental delay and autism, respectively. This study aimed to evaluate the effectiveness of implementing 18-month developmental and autism screening by mail, with a standardized follow-up process for abnormal screen results. METHODS: Within a prospective cohort study design, parents of 892 children received by mail the 18-month ASQ and M-CHAT questionnaires between December 2008 and September 2009. A registered nurse scored the questionnaires and, if needed, administered follow-up screening or set up a referral to Early Intervention or subspecialty pediatrics. Medical record reviews determined clinical outcomes of children who required intervention after initial screening through September 2010. Additionally, demographic factors were evaluated for association with responding to the questionnaires. RESULTS: The ASQ and M-CHAT questionnaires were returned by 529 (59.3%) of the parents. Parents of White children (390/575 [67.8%]) and those with private insurance (457/660 [69.2%]) were significantly more likely to return screening questionnaires than parents of non-White (64/171 [37.4%]; P < .001) and government-insured children (58/169 [34.3%]; P < .001), respectively. Of the 529 children with returned surveys, 109 (20.6%) did not pass at least 1 of the initial screens and 12 (2.3%) were referred after not passing the follow-up screening process. CONCLUSIONS: Developmental and autism screening by mail is not a sufficient method to comprehensively screen a general pediatric population. A nurse-completed, standardized follow-up process after an initial failed developmental screen may increase the yield of appropriate Early Intervention or subspecialty referrals.